Improving Deep Mutual Learning via Knowledge Distillation

Knowledge transfer has become very popular in recent years, and it is either based on a one-way method used with knowledge distillation or two-way implemented by deep mutual learning, while both of them adopt teacher–student paradigm. A more simple compact because only involves an untrained low-capacity student high-capacity teacher network the process. In contrast, requires training costs two low-cost capacities from scratch simultaneously to obtain better accuracy results for each network. this paper, we propose new approaches, namely full learning (FDDML) half (HDDML), improve convolutional neural performance. These approaches work three losses using variations existing architectures, experiments have been conducted public benchmark datasets. We test our some KT task methods, showing its performance over related methods.